Skip to content

Fix fsspec data loss after flush and reduce stream buffering costs - #407

Open
shcheklein wants to merge 2 commits into
mainfrom
fix/fsspec-stream-io
Open

shcheklein wants to merge 2 commits into
mainfrom
fix/fsspec-stream-io

Conversation

@shcheklein

@shcheklein shcheklein commented Nov 22, 2026 •

Copy link
Copy Markdown
Member

On main, calling flush() on a Drive writer uploads the bytes written so far and marks the stream closed. A later binary write() still accepts bytes, but close() skips the final upload. For example, writing b"first", flushing, writing b"last", and closing leaves b"first" on Drive.

This change makes flush() flush the local buffer and close() upload once. Binary writes return their byte count. Small files stay in memory and larger files spill to a seekable temporary file. Upload errors propagate on close and the spool is released. Callers that need the file to be visible on Drive should close the writer or exit its context manager.

Small reads previously copied the entire unread tail of the current Drive response. The reader now keeps a view of that tail. A positive block_size passed to fs.open() controls the size of Drive download requests.

Local measurements

On macOS with CPython 3.11.18, comparing this branch with 4798a5c. The read result is the median of seven runs. The write measurement uses tracemalloc around the writer, reuses one input block, and excludes network traffic.

Workload main This PR
Read one 8 MiB response in 8 KiB pieces 193.85 ms 0.70 ms
Stage 64 MiB in 1 MiB writes, peak Python allocation 64.13 MiB 4.01 MiB

Verification

The offline suite passed locally (138 passed, 2 skipped). The filesystem unit tests passed on fsspec 2021.7.0 and 2026.9.0. Coverage includes flush followed by another write in text and binary mode, disk rollover, cleanup after a failed upload, byte-range request sizes, and mixed partial reads and peeks. The existing live Drive round-trip test now flushes between two writes and checks the downloaded bytes. Ruff, mypy, and the Sphinx build passed.

Based on main. Git verified a clean merge with the separate protocol-registration PR #406.

Comment thread pydrive2/test/test_fs_unit.py Fixed
Comment thread pydrive2/test/test_fs_unit.py Fixed

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

There are no active review issues for the Drive fsspec write-handling and buffering changes.

No open issues 🎉

What changed in this PR

This PR fixes data loss when writing after flush() through the Drive fsspec integration and reduces stream buffering overhead.

Changes:

  • Keep writes local until close, using temporary-file spooling for larger uploads and propagating upload errors.
  • Avoid copying unread response tails and honor positive download block_size values.
  • Add regression coverage and document flush, close, and buffering behavior.
File Description
pydrive2/​test/​test_fs.py Adds a flush between writes in the live Drive round-trip test.
pydrive2/​test/​test_fs_unit.py Tests partial reads, peeks, download sizes, writer rollover, and upload-failure cleanup.
pydrive2/​fs/​utils.py Uses memory views for unread data and preserves buffered bytes after peek failures.
pydrive2/​fs/​spec.py Adds spooled writes, upload-on-close behavior, and configurable download request sizes.
docs/​fsspec.rst Documents download sizing and local buffering versus upload behavior.

Review effort: Balanced

@shcheklein shcheklein left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI-generated review

Overall this is a clean, well-scoped fix with strong test coverage and impressive memory/latency improvements. The flush()/close() lifecycle and IterStream memoryview slicing are much cleaner than main.

Three edge-case bugs found (see inline comments):

  1. Temp-file leak on write() after close() in GDriveBufferedWriter.write() (pydrive2/fs/spec.py).
  2. TypeError on block_size="default" in _gdrive_open_file() (pydrive2/fs/spec.py), which is a standard fsspec sentinel.
  3. memoryview export left open on exception in IterStream.readinto() (pydrive2/fs/utils.py), preventing callers from resizing/clearing a bytearray inside an except block.

Comment thread pydrive2/fs/spec.py Resolved
Comment thread pydrive2/fs/spec.py Resolved
Comment thread pydrive2/fs/utils.py Resolved

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants